Also spoke with orange about uploading SODA test data:
First, if you're not familiar with Rucio, when you upload to the datalake you need to specify a "scope" (or namespace) where the data will live, and a unique "name" within this namespace. Combining these gives you a unique identifier, or "DID" with the format scope:name. Be careful with the name choice, it cannot be reused ever within the namespace. Previously we have prefixed the name of files with the datetime, that is, the output of something like date "+%F-%H-%M-%S" to make sure there aren't name conflicts.
I have set up two scopes for you to put data in. These are orange and sp3531_soda. orange is a scratch or throwaway namespace, i.e. you can put any unimportant ephemeral data in here. sp3531_soda is reserved for your test files related to feature SP3531.
There are several commands you will need to begin with. The first is the rucio upload commands. You can test an upload with something like:
[rbarnsley@src-dev ~]$ export FILE=`uuidgen` && echo "test">> $FILE && rucio upload --rse SPSRC_STORM --lifetime 3600 --scope orange --register-after-upload $FILE
2023-06-29 08:15:44,818 INFO Trying upload with https to SPSRC_STORM
360
2023-06-29 08:15:46,127 INFO Successful upload of temporary file. https://spsrc14.iaa.csic.es:18027/disk/dev/deterministic/orange/53/d8/b1a51c96-2200-4f32-ad90-d985456d7d85.rucio.upload
2023-06-29 08:15:46,639 INFO Successfully uploaded file b1a51c96-2200-4f32-ad90-d985456d7d85
2023-06-29 08:15:48,330 INFO Successfully added replica in Rucio catalogue at SPSRC_STORM
2023-06-29 08:15:48,374 INFO Successfully added replication rule at SPSRC_STORM
This creates a file with a unique name $FILE (generated with uuidgen), puts "test" into it and uploads it. The parameters to this call are:
--rse: specify a storage element to upload to, in this example, we're using the Spanish SRC (SPSRC_STORM)
--lifetime: the lifetime of the file in seconds. This is critical, otherwise the file will be stored indefinitely
--scope: the scope the uploaded file will live in (in this case, i've used orange)
--register-after-upload: tells Rucio to only reserve the name after a successful upload. This is important as without it, even if an upload fails, the name will not be allowed to be reused.
After the upload completes, a "rule" will be created that tells Rucio to keep this file stored at a particular storage site (in the example above, this site is SPSRC_STORM). You can view these rules with, e.g.
[rbarnsley@src-dev ~]$ rucio list-rules orange:$FILE
ID ACCOUNT SCOPE:NAME STATE[OK/REPL/STUCK] RSE_EXPRESSION COPIES EXPIRES (UTC) CREATED (UTC)
-------------------------------- ----------- ------------------------------------------- ---------------------- ---------------- -------- ------------------- -------------------
f759af67027846d0a7986a41b625cbf5 robbarnsley orange:b1a51c96-2200-4f32-ad90-d985456d7d85 OK[1/0/0] SPSRC_STORM 1 2023-06-29 09:15:48 2023-06-29 08:15:48
You can also view the resulting DID with:
[rbarnsley@src-dev ~]$ rucio list-dids orange:$FILE --filter "type==file"
+---------------------------------------------+--------------+
| SCOPE:NAME | [DID TYPE] |
|---------------------------------------------+--------------|
| orange:b1a51c96-2200-4f32-ad90-d985456d7d85 | DIDType.FILE |
+---------------------------------------------+--------------+
So that's upload. The next bit you're going to need is to modify metadata. To set metadata you need the rucio set-metadata command. Likewise, there is a rucio get-metadata command. An example with the file above is:
[rbarnsley@src-dev ~]$ rucio set-metadata --did orange:$FILE --key some_key --value some_value
[rbarnsley@src-dev ~]$ rucio get-metadata --plugin POSTGRES_JSON orange:$FILE
some_key: some_value
the --plugin POSTGRES_JSON is important when retrieving metadata - this tells it to retrieve it from the external store.
Now, if you set the following IVOA keys in the metadata:
calib_level
obs_collection
obs_id
obs_publisher_did
then, in theory (!), your record should also be propagated through to our IVOA services, e.g. scs. Other metadata keys with the same name as obscore fields will also be propagated, but the ones above are mandatory for the record to get created.
To use our datalink, you must set the access_url to point to our service with query parameter "id" equal to the DID, e.g. for the above file
[rbarnsley@src-dev ~]$ curl https://ivoa.datalink.srcdev.skao.int/rucio/links?id=orange:b1a51c96-2200-4f32-ad90-d985456d7d85
<?xml version="1.0" encoding="UTF-8"?>
<VOTABLE xmlns="http://www.ivoa.net/xml/VOTable/v1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
version="1.4">
<RESOURCE type="results">
<TABLE>
<FIELD name="ID" datatype="char" arraysize="*" ucd="meta.id;meta.main" />
<FIELD name="access_url" datatype="char" arraysize="*" ucd="meta.ref.url" />
<FIELD name="service_def" datatype="char" arraysize="*" ucd="meta.ref" />
<FIELD name="error_message" datatype="char" arraysize="*" ucd="meta.code.error" />
<FIELD name="semantics" datatype="char" arraysize="*" ucd="meta.code" />
<FIELD name="description" datatype="char" arraysize="*" ucd="meta.note" />
<FIELD name="content_type" datatype="char" arraysize="*" ucd="meta.code.mime" />
<DATA>
<TABLEDATA>
<TR>
<TD>ivo://test.skao/~?orange:b1a51c96-2200-4f32-ad90-d985456d7d85</TD>
<TD>['https://spsrc14.iaa.csic.es:18027/disk/dev/deterministic/orange/53/d8/b1a51c96-2200-4f32-ad90-d985456d7d85']</TD>
<TD />
<TD />
<TD>#this</TD>
<TD></TD>
<TD>image/fits</TD>
</TR>
</TABLEDATA>
</DATA>
</TABLE>
</RESOURCE>
</VOTABLE>
and set access_format accordingly (application/x-votable+xml;content=datalink).Adding authentication via bearer token to API REST skeleton.
Connected DaCHS/datalink to a binder hosted notebook.
Looking at OIDC for rest skeleton using only CLI.
Looking over API stuff.
Trying to get postgres metadata db to work on minikube:
$ kubectl create -f /tmp/pv.yaml
$ kubectl create -f /tmp/pv.yaml; helm upgrade --install postgres-metadata --namespace rucio-ivoa-integration bitnami/postgresql --values values.yaml --set volumePermissions.enabled=true
$ alias k="kubectl -n rucio-ivoa-integration"
$ helm delete --namespace rucio-ivoa-integration postgres-metadata & k delete pvc data-postgres-metadata-postgresql-0 & k delete pv pv; Turns out needed to add pg_hba to the helm values rather than bake it in to the Dockerfile (gets overwritten).
Added ivoa.dachs and ivoa.datalink routes to proxy. Need to update ansible on infra machine.
Added dachs image and bitnami-dachs postgres image to container registry. Created helm charts for this (postgres uses base bitnami image and helm chart).
Finished creating new CLI for rucio-extended-client.
Tidied dashboard up a bit.
Started amending rucio-extended-client (adding bits in for subparsers).
Looked at how it may be possible to get PFN from web uri for SODA. Can't just be linked to technology. For WebDAV, maybe propfind?
PI 19 planning.